Recording of 3D images of a scene with phase de-convolution

ABSTRACT

A method of recording 3D images of a scene based on the time-of-flight principle is described. The method includes illuminating a scene by emitting light carrying an intensity modulation, imaging the scene onto a pixel array using an optical system, detecting, in each pixel, intensity-modulated light reflected from the scene onto the pixel and determining, for each pixel, a distance value based on the phase of light detected in the pixel. The determination of the distance values includes a phase-sensitive de-convolution of the scene imaged onto the pixel array such that phase errors induced by light spreading in the optical system are compensated for.

TECHNICAL FIELD

The present invention generally relates to a method of recording 3Dimages of a scene based upon the time-of-flight measurement principleand to a 3D imager configured for implementing the method.

BACKGROUND

The time-of-flight measurement principle is well known in the field of3D imaging. 3D cameras (or range cameras) are known that acquire rangeimages in real time based on the time-of-flight (TOF) principle. Suchcamera generally comprises a light source emitting sinusoidallymodulated light into the scene to be imaged in 3D and a pixel array onwhich the scene is imaged by an optical system. The camera thencorrelates the light detected in the pixels with the light emitted anddetermines, for each pixel, the phase difference between emitted andreceived light. This phase difference is proportional to the radialdistance between the camera and the part of the scene that is imagedonto the pixel concerned. As the demodulation is synchronously performedfor all pixels of the pixel array, the camera provides an array ofdistance values associated each to a particular pixel and thus to aparticular part of the scene. In the following, we will also use “phase”instead of “phase difference”; it is understood that the phase of theemitted light or a clock signal, used for modulating the emitted lightor derived from the modulation of the emitted light, then serves as areference phase. It should also be noted that, as used herein, “phase”and “phase difference” always refer to the phase of the modulation, notto the phase of the carrier wave that is modulated.

The demodulation process, which leads to the determination of the phaseof the light impinging on the pixels, can be carried out in differentways. EP 0 792 555 discloses a 3D camera with a one- or two-dimensionalpixel array, each pixel thereof comprising a light-sensitive part, inwhich charge carriers are generated in response to light impingingthereon, and a light-insensitive part with a plurality of electricalswitches and storage cells associated with a respective one of theseswitches. The charges that are integrated in the light-sensitive partare transferred to the storage cells by sequential actuation of theelectrical switches. The electrical switches are controlled in such away that the charges transferred to a particular storage cell belong toa time interval or time intervals at a known phase of the emitted light.The charges accumulated in the different storage cells are then used todetermine the phase of the light having impinged on the pixel, itsamplitude and a background light level. More details on that principleof measurement can be found in the paper “The Lock-InCCD—Two-dimensional Synchronous Detection of Light” by Spirig et al. inIEEE Journal of Quantum Electronics 31 (1995), 1705-1708. An improvementof this method of demodulation is described in EP 1 659 418.

U.S. Pat. No. 6,825,455 discloses another way for demodulating thedetected light. In this document, the light-sensitive part of each pixelcomprises at least two modulation photogates and the light-insensitiveregion comprises accumulation gates, each of which is associated to arespective modulation photogate. During a first exposition interval,charge carriers generated in the light-sensitive part of the pixel inresponse to light impinging thereon are exposed to a first voltagegradient modulated at the frequency of the modulation of the emittedlight and thereby caused to drift into a first accumulation gate whenthe voltage is of a first polarity and into a second accumulation gatewhen the voltage is of the opposite polarity. The charges q_(a) andq_(b) so accumulated in the first and second modulation gates,respectively, are determined. During a second exposition interval,charge carriers generated in the light-sensitive part of the pixel areexposed to a second voltage gradient modulated at the same frequency butshifted by a known phase with respect to the first voltage gradient. Thecharge carrier are again caused to drift into two different accumulationgates in accordance with the polarity of the voltage applied, givingrise to accumulated charges q_(c) and q_(d). The phase of the lightimpinging on the pixel is determined using the values of the accumulatedcharges. If the phase difference between the voltage gradients amountsto 90°, the phase of the light can be determined as φ=arctan[(q_(c)−q_(d))/(q_(a)−q_(b))]. Above-cited documents are herewithincorporated herein by reference in their entirety.

For the sake of comprehensibility of the invention, we will brieflyrecall the basic mathematical concept of the measurement according tothe TOF principle in a 3D camera working with continuously modulatedlight.

An illumination unit of the camera emits a continuously modulated lightintensity that can be described by the formula:S(t)=S ₀·(1+sin(ωt))  (1)

where S₀ is the average light intensity and ω is given by the modulationfrequency f, i.e. ω=2πf. The scene is thus continuously illuminated witha light power density P that depends on the illumination strength S, thespatial distribution of the light and the distance between scene andcamera. A part of the light power, given by a remission coefficient ρ,is then remitted by the objects in the scene. As used herein,“remission” designates reflection or scatter of light by a material. Theimager optics maps the remitted light that passes through the opticalsystem (e.g. comprising one or more lenses and/or prisms and/or filters,etc.) onto the pixel array of the camera. Assuming an ideal opticalsystem, the received light intensity I(x,t) that arrives at time t onpixel position x=(u,v) of the pixel array thus has the followingcharacteristics:

The light intensity I(x,t) is modulated in time with the same frequencyas the emitted light, however, with the phase retarded by a value φproportional to the distance r between the camera and the part of thescene that is mapped to point x=(u,v) on the imager. Mathematically, thereceived light intensity is thus given by the formulasI(x,t)=B(x)+A(x)sin(ωt−φ(x))  (2)andφ(x)=2r(x)ƒ/c  (3)with c denoting the speed of light, A the amplitude of the modulation ofthe received light and B (>A) the constant offset of the modulated lightand background light originating from other light sources illuminatingthe scene (e.g. the sun). One assumes here that A, B and φ are at mostslowly varying, so that they may be regarded as constant on thetimescale of the modulation.

The amplitude A is thus proportional to the power density P on the partof scene that is mapped onto the corresponding pixel by the opticalsystem, the remission coefficient ρ of that part of the scene andparameters of the optical system that are independent on the lightpower, like the F-number of the lens.

An ideal optical system maps a point in the scene onto a point in theimage plane. In practice, however, light originating from a point in thescene is spread on an area around the theoretical image point. Variousphysical mechanisms may cause such spread of the image point. Defocusingof the lens causes a locally confined spread area that makes an image toappear unsharp. The relation between sharpness, defocusing and thespread area is described by the concept of depth of field. Othermechanisms leading to a point spread are light diffraction (in case of asmall aperture of the lens), multiple light scattering on surfaces ofthe optical system or light scattering due to a contamination of thesurface of the optical system. These physical effects lead to a loss ofcontrast in the image.

The effect of light spreading of a point source can be describedmathematically by the so-called point spread function (PSF). If x=(u,v)and x′=(u′,v′) define two points in the image plane, the value g(x′,x)of the PSF function g indicates the relative amount of light that ismapped onto point x′ when the theoretical image point is x. Due to thesuperposition principle (that is valid for linear systems like anoptical imaging system), the effect of the light spreading onto an imagecan be described by a convolutionI′=g*I  (4a)that isI′(x)=∫g(x, x′)I(x′)dx′,  (4b)where I denotes the ideal image and I′ the actual image affected bylight spreading in the optical system. If an image is given as discretepoints (pixels) the integral represents a sum over the pixels.

In order to reconstruct the ideal image I from an image I′ provided bythe optical system, convolution (4a) or (4b) has to be inverted. Thisinversion is called a de-convolution and is in the ideal case realizedby convolution of the image I′ with a de-convolution function g′ (whichfulfils, at least approximately, the condition that it's convolutionwith g is the Dirac-delta function). Such a de-convolution function is,however, not known in general and also not always uniquely defined. Astandard approach toward de-convolution is based on the transformationof the image and the convolution function to Fourier space. However,this approach is not always applicable. In the field of image processingvarious approaches have been developed for de-convoluting an image atleast approximately.

In 2D imaging, light spreading is often negligible under normalconditions (using ambient light and well-focused optics). The pointspread function g(x′,x) is then close to a delta peak, e.g.:g(x′,x)=(1−ε)δ(x′,x)+εƒ(x′,x)  (5)where ƒ is a function normalized to 1 and ε the relative amount of lightscattered in the optical system. In a well-focused and clean opticalsystem, ε is typically small e.g. of the order 10⁻³. A blur visible forthe human eye therefore occurs only if light from a very bright lightsource (e.g. the sun) is shining into the optical system. In this case,the contribution of the spread light from the bright light source cannotbe neglected, since its intensity is many orders of magnitude higherthan the light intensity reflected by an object in the scene. If theoptical system is contaminated (with dirt or a scratch), the parameter εis larger, so that light spreading visible for the human eye couldresult even in normal lighting conditions.

The inventors have recognised that in a TOF camera system broadlyilluminating the scene, an effect similar to image blurring due toscattered sun light can occur due to the active illumination. The mainreason is that the light power density P on an object in the scenestrongly depends on the distance d of the object to the light source(P˜1/d²). The light intensity I′(x) at the corresponding pixel positionx is proportional to the light power density and the remissioncoefficient ρ, i.e.I′(x)˜ρ/d ²  (6)

As an example, an object with a remission coefficient of 50% at adistance of 1 m will generate an intensity value, which is 1000 timeslarger than the intensity value generated by an object with a remissioncoefficient of 5% at a distance of 10 m. Therefore, when (5) issubstituted into the convolution integral (4b), the contributions ofintensities at points x≠x′ are no longer negligible, even if the lightscattering factor ε is of the order 10⁻³.

It is important to recognise that the phase measurement and thus thecomputed distance information is falsified by light spreading. This willnow be explained in more detail first for a superposition of twomodulated light intensities and then for the general case.

The superposition of two modulated light intensities expressible byequation (2) yields:I′(t)=I ₁(t)+I ₂(t)=(B ₁ +B ₂)+A ₁ sin(ωt−φ ₁)+A ₂ sin(ωt−φ ₂)  (7a)I′(t) can again be expressed in the form of one modulated lightintensity, i.e.I′(t)=B′+A′ sin(ωt−φ′)  (7b)whereB′=B ₁ +B ₂A′=√{square root over (AS′ ² +AC′ ²)}φ′=arctan(AS′/AC′)  (8)withAS′=A ₁ sin φ₁ +A ₂ sin φ₂ =:AS ₁ +AS ₂AC′=A ₁ cos φ₁ +A ₂ cos φ₂ =:AC ₁ +AC ₂.  (9)

Formulas (7a) to (9) show that the superposition of two modulatedintensities with same frequency but different phases and amplitudesresults in a modulated intensity with again the same frequency but whosephase depends not only on the phases but also on the amplitudes of theindividual intensities being superposed. In other words, light spreadingin presence of a non-ideal optical system induces errors in the measuredphase values.

Before turning to the general case of superposition of modulatedintensities due to spreading, it shall be observed that is convenient torewrite equation (9) in complex notation:Â′:=A′e ^(iφ′) =A ₁ e ^(iφ) ¹ +A ₂ e ^(iφ) ² =:Â ₁ +Â ₂  (10)where AC and AS are the real and the imaginary components, respectively,of the complex amplitude Â, i.e.AS′=ImÂ′AC′=ReÂ′  (11)

The superposition principle (9) or (10) for the amplitudes can bestraightforwardly generalized for the case that the optical systemspreads the light intensity with a point spread function g. UsingI(t)=B+A sin(ωt−φ)=B−Im(A·e ^(i(φ-ωt)))=B−Im(Âe ^(−iωt))  (12)andI′(t)=B′+A′ sin(ωt−φ′)=B′−Im(A′·e ^(i(φ′-ωt)))=B′−Im(Â′e ^(−iωt))  (13)and substituting this into equation (4b), the result isÂ′(x)=∫g(x, x′)Â(x′)dx′.  (14)

The resulting phase φ′(x) and amplitude A′(x) are again given byequation (8) using the real and imaginary parts of Â′(x) as defined in(11).

As a result of the non-negligible superposition, the contrast in phasemeasurement is reduced. This means that the phases measured in thedifferent pixels are shifted towards the phase of the pixel with thestrongest amplitude of modulation. This effect of phase shift is thestronger, the smaller the amplitude of the corresponding pixel is.Therefore, the phase shift caused by light spreading affects mostlybackground pixel. The objects in the background of the scene appear thusnearer to the camera than they actually are, especially if thebackground part of the scene has a low remission coefficient.

The invention generally seeks to reduce the effect of light spreadingonto the range measurement.

BRIEF SUMMARY

The method of recording 3D images of a scene comprises the followingsteps:

-   -   illuminating a scene by emitting light carrying an intensity        modulation;    -   imaging the scene onto a pixel array using an optical system;    -   detecting, in each pixel, intensity-modulated light reflected        from the scene onto the pixel;    -   and determining, for each pixel, a distance value based on the        phase of light detected in the pixel.

According to the invention, the determination of the distance valuescomprises a phase-sensitive de-convolution of the scene imaged onto thepixel array such that phase errors induced by light spreading in theoptical system are compensated for. As indicated above, de-convolutingis a known measure for at least partially compensating the effect oflight speading in a 2D imager, i.e. an imager not providing depth (ordistance or range) information. However, to the knowledge of theinventors, the problem of loss of phase contrast in a 3D imager due tolight spreading has not yet been addressed in the literature. The methodproposed herein permits the detection of more accurate 3D images.Furthermore, the method enables accurate distance determination in moredifficult lighting conditions.

A first embodiment of the method is especially suitable for a 3D imager,which outputs, for each pixel an amplitude value and a phase value(optionally also the constant background intensity but this isirrelevant for the further considerations) of the light impinging on thepixel, including stray light if any. Thus, according to the firstembodiment of the invention, for each pixel, an amplitude value (A′(x)using the above notation) and a phase value (φ′(x)) of theintensity-modulated light detected in the pixel are determined. Thephase-sensitive de-convolution of the scene imaged onto the pixel arraycomprises:

-   -   forming a first data array, each array element of the first data        array being associated with a respective pixel of the pixel        array and having a value corresponding to the amplitude value        determined for the associated pixel, possibly weighted with a        first phase factor;    -   forming a second data array, each array element of the second        data array being associated with a respective pixel of the pixel        array and having a value corresponding to the amplitude value        determined for the associated pixel weighted with a second phase        factor, the second phase factor depending on the phase value        determined for the associated pixel;    -   de-convoluting the first and second arrays using a        de-convolution function of the optical system.

For each pixel, the distance value then is calculated based upon thevalues of the array elements of the de-convoluted first and secondarrays associated to the pixel.

Those skilled will appreciate that first phase factor is preferably thecosine of the phase value determined for the associated pixel (i.e.cos(φ′(x))) and the second phase factor is preferably the sine of thephase value determined for the associated pixel (i.e. sin(φ′(x))). Theelements of the first data array then take the valuesAC′(x)=A′(x)·cos(φ′(x))  (15)and those of the second data array take the valuesAS′(x)=A′(x)·sin(φ′(x)).  (16)

Alternatively, the first and second phase factors could beapproximations of the cosine and the sine of the phase value,respectively. For instance, using the so-called small-phaseapproximation. We will first assume that |φ′(x)|<<2π. In this case, thefirst phase factor may be set equal to 1 and the determined phase φ′(x)itself may serve as approximation of the sine of the phase. In thiscase, the elements of the first data array then take the valuesAC′(x)=A′(x)  (17)and those of the second data array take the valuesAS′(x)=A′(x)·φ′(x).  (18)

In both cases, the elements of the de-convoluted versions of the firstand second data arrays may then be evaluated by:AC(x)=∫g′(x, x′)AC′(x′)dx′  (19a)andAS(x)=∫g′(x, x′)AC′(x′)dx′  (19b)where g′ is the de-convolution function of the optical system. g′ may begiven by a matrix if the integral is a sum over all pixels of the pixelarray. Equations (19a) and (19b) may be summarised asÂ(x)=∫g′(x, x′)Â′(x′)dx′.  (20)

For each pixel, the corresponding distance value can be calculated byevaluating a corrected phase asφ(x)=arctan(AS(x)/AC(x)),  (21)or, if the small-phase approximation is used, asφ(x)=AS(x)/1 AC(x).  (22)

It shall be noted that the small-phase approximation may also be used ifthe phases of the different pixels φ′(x) do not necessarily satisfy thecondition |φ′(x)|<<2π but lie within a relatively narrow range. One maythen write φ′(x)=φ₀′+δφ′(x) with |δφ′(x)|<<2π, where φ₀′ is an offsetcommon to all the pixels of the array, e.g. the average of the measuredphase values φ′(x) or a predetermined constant. In this case, one mayuse δφ′(x) instead of φ′(x) in equations (17) and (18). Thede-convolution of the first and second data arrays is achieved throughequations (19a) and (19b). One finds the corrected phase φ(x) usingδφ(x)=AS(x)/AC(x) and φ(x)=δφ(x)+φ₀. Those skilled will note that thesubtraction and the latter addition of φ₀ corresponds to a change of thereference phase which may be chosen arbitrarily; therefore, in thefollowing it will be assumed that the reference phase is chosen suchthat φ₀=0 (thus φ′(x)=δφ′(x)) when the small-phase approximation isused.

If it is desired to express the distance in other units than units ofphase, this may be done by using equation (3).

According to a second preferred embodiment of the method, the detectionof intensity-modulated light reflected from the scene comprises, foreach pixel, determining intensity values of the intensity-modulatedlight impinging on the pixel at different phases of the modulation, thedifferent phases being chosen such that amplitude and phase of theintensity-modulated light impinging on the pixel are derivable from theset of intensity values using a known relationship. However, as thoseskilled will appreciate, in the second embodiment it is not necessarythat the uncorrected amplitude and phase values have actually beencalculated using the known relationship. In this case, thephase-sensitive de-convolution of the scene comprises

-   -   forming data arrays, each array element of the data arrays being        associated with a respective pixel of the pixel array and having        a value corresponding either to the intensity value of the        associated pixel determined at one of the phases of modulation        or to a linear combination of at least two intensity values of        the associated pixel determined at different phases of the        modulation;    -   de-convoluting the data arrays using a de-convolution function        of the optical system;

For each pixel, the distance value is then calculated based upon thevalues of the array elements of the de-convoluted data arrays associatedto the pixel, e.g. by determining the corrected phase from the values ofthe array elements of the de-convoluted data arrays associated to thepixel.

For each pixel, the actually determined light intensity I′(x) can bemathematically expressed as a function of time according to equation(7b) or (13) with a priori unknown parameters A′(x), B′(x) and φ′(x). Toallow a determination of these parameters, the intensity values of eachpixel thus have to be determined at least three different phases of themodulation, as explained in detail in the paper by Spirig cited above oraccording to the approach of U.S. Pat. No. 6,825,455 (where the chargesq_(a), q_(b), q_(b) and q_(d) correspond to the intensity values atdifferent phases).

Preferably, the at least three phases of the modulation are regularlyspaced; Most preferably, the intensity values are determined at fourphases of the modulation, these four phases of the modulation beingspaced by 90 degrees. In the latter case, the four data arrays may haveas elements the intensity values I₀′(x) associated to the 0°-phase ofthe modulation, I₁′(x) associated to the 90°-phase of the modulation,I₂′(x) associated to the 180°-phase of the modulation and I₃′(x)associated to the 270°-phase of the modulation, respectively.

These data arrays are then de-convoluted using the de-convolutionfunction of the optical system, yielding the de-convoluted data arrayshaving as elements:I _(k)(x)=∫g′(x, x′)I _(k)′(x′)dx′  (23)where k=0, 1, 2 and 3, respectively. The corrected phase may then becalculated for each pixel based upon the corrected intensity valuesI₀(x), I₁(x), I₂(x) and I₃(x) using

$\begin{matrix}\begin{matrix}{{{AS}(x)} = {{{A(x)} \cdot {\sin\left( {\varphi(x)} \right)}} = {\frac{1}{2}\left( {{I_{2}(x)} - {I_{0}(x)}} \right)}}} \\{{A\;{C(x)}} = {{{A(x)} \cdot {\cos\left( {\varphi(x)} \right)}} = {\frac{1}{2}\left( {{I_{1}(x)} - {I_{3}(x)}} \right)}}}\end{matrix} & (24)\end{matrix}$and equation (21).

Instead of de-convoluting data arrays, the array elements of which areeach associated with specific phase of the modulation and have a valuecorresponding to the intensity value of the associated pixel determinedat one of the phases of modulation as in equation (23), one mayalternatively de-convolute data arrays having as array elements linearcombinations of at least two intensity values of the associated pixeldetermined at different phases of the modulation, e.g.AS′(x)=(I₂′(x)−I₀′(x))/2 and AC′(x)=(I₁′(x)−I₃′(x))/2. Thede-convolution may in this case be effected by computing AS and ACaccording to equations (19a) and (19b).

According to an advantageous aspect of the invention, a level ofcontamination of the optical system is evaluated and the phase-sensitivede-convolution is adjusted to the level of contamination. The evaluationof the level of contamination is preferably achieved as explained inEuropean patent application 07 110 379.0.

Another aspect of the invention concerns a 3D imager configured forimplementing the method as described before. Such 3D imager (e.g. a 3Dcamera) may comprise a light source for illuminating a scene by emittinglight carrying an intensity modulation, an optical system, an array ofpixels configured for detecting intensity-modulated light reflected fromthe scene and imaged onto the pixel array, and a control and evaluationcircuit configured for determining, for each pixel, a distance valuebased on the phase of light detected in the pixel. The control andevaluation circuit comprises means, e.g. an application-specificintegrated circuit (ASIC), a field-programmable gate array (FPGA) and/ora microprocessor, for carrying out a phase-sensitive de-convolution ofthe scene imaged onto the pixel array in such a way as to compensate forphase errors induced by light spreading in the optical system.

BRIEF DESCRIPTION OF THE DRAWINGS

Further details and advantages of the present invention will be apparentfrom the following detailed description of several not limitingembodiments with reference to the attached drawings, wherein:

FIG. 1 is a schematic view of a 3D camera operating according to theTOF-principle;

FIG. 2 is an illustration of how the parameters of the intensitywaveform are affected in the presence of light spreading;

FIG. 3 is a flow chart of the method according to the first embodimentof the invention;

FIG. 4 is a flow chart of the method according to the second embodimentof the invention.

DETAILED DESCRIPTION

FIG. 1 shows a 3D camera generally identified by reference numeral 10.The 3D camera 10 comprises an illumination unit 12, for emittingsinusoidally modulated light into a scene, a two-dimensional pixel array14 and an optical system 16 (represented here by a lens) imaging thescene onto the pixel array 14. The pixel array 14 may be implemented asan electronic camera chip of any suitable technology, such as CCD, CMOSand/or TFA. The pixel array comprises individual lock-in pixel sensorcells 18 (herein simply called pixels), on each of which a small portionof the scene is imaged.

The illumination unit 12 may comprise one or several individual lightemitting devices, e.g. light emitting diodes (LEDs), which arecollectively driven by an illumination driver 20. A clock signal source22 (e.g. a numerically controlled oscillator) provides the input signalsfor the illumination driver 20 and the photo gate driver 24, whichcontrols the pixel array 14. An evaluation circuit 26 (e.g. an ASIC, anFPGA, or a digital signal processor (DSP)), connected to the pixel array14, determines, when the 3D camera is operating, the distanceinformation based upon the charges generated in the pixels.

When the 3D camera 10 is in operation, the signal source 22 generates amodulation signal on its output and feeds this modulation signal to theillumination driver 20. The latter drives the illumination unit 12 witha drive signal, thereby causing the illumination unit to emit light(indicated by the dash-dotted lines 13) carrying a sinusoidal intensitymodulation into the scene. For purpose of illustration, the scene isrepresented here as comprising a foreground object 28 and a backgroundobject 30. It should be noted that the drawing is not to scale and thedistance between the camera 10 and the objects 28, 30 in the scene ispreferably substantially larger than the distance between theillumination unit 12 and the optical system 16 (which are preferablyintegrated within a single housing). The modulated light is remitted(reflected or scattered) by the objects 28, 30 and a fraction of theremitted light is received by the pixel array 14. The signal source 22also feeds the modulation signal to the photo gate driver 24 whichcontrols the individual pixels 18 so that they operate, for instance, asdescribed hereinbefore with reference to EP 0 792 555 or U.S. Pat. No.6,825,455.

In the following, we will assume that the pixel array 14 and the photogate driver 24 are configured so as to operate according to theprinciples of the former document. In this case, each pixel 18 comprisesa light-sensitive part, in which charge carriers are generated inresponse to light remitted from the scene impinging thereon, and alight-insensitive part with a plurality of electrical switches andstorage cells associated with a respective one of these switches. Thefollowing considerations of this paragraph are with respect to anindividual pixel. The charges that are integrated in the light-sensitivepart are transferred to the storage cells by sequential actuation of theelectrical switches under the control of the photo gate driver 24. Theelectrical switches are controlled in such a way that the chargestransferred to a particular storage cell belong to a time interval ortime intervals at a known phase of the emitted light. FIG. 2 illustrateshow these integration intervals may be distributed in one period of themodulation. The light intensity I′(x,t) impinging on the pixel isindicated by the dashed curve 32. Mathematically, it may be expressed byequations (7b) or (13), where the parameters A′(x), B′(x) and φ′(x) arenot known from the beginning. The charge carriers generated in thelight-sensitive part of the pixel during a first time interval T₁ aretransferred to a first storage cell by closing the correspondingelectrical switch at a specific time. After that transfer, the firstelectrical switch is opened again and after a specific time interval haselapsed, the charge carriers generated during a second time interval T₂are transferred to the second storage cell by closing the secondelectrical switch at a specific time. The same process is repeated forthe time intervals T₃ and T₄. It is possible to extend the process overseveral periods of the modulation. The charges accumulated in thedifferent storage cells thus correspond to intensity values of themodulated light at different phases of the modulation and may be used todetermine the parameters A′(x), B′(x) and φ′(x) of the light havingimpinged on the pixel. With respect to the timescale of the modulation,A′(x), B′(x) and φ′(x) (and thus the corrected parameters A(x), B(x) andφ(x)) are slowly varying. A typical modulation period is, for instance,50 ns, which corresponds to a 20-MHz modulation or a camera range of 7.5m. In applications such as occupancy detection of a vehicle seat, theabove parameters may be regarded as practically constant over severalmodulation periods.

One or more of the pixels 18 (e.g. an entire row) of the pixel array areused as reference pixels 19. Light emitted by the illumination unit 12is guided onto the reference pixels 19 using a light guide 15 (e.g. anoptical fibre or a bundle of optical fibres) of known length. Theintensity values retrieved from the reference pixel indicate a referencephase having a known offset with respect to the phase of the modulatedlight at the emission by the illumination unit 12. The reference pixels19 are appropriately protected from light remitted from the scene toavoid distortion of the reference phase.

As illustrated in FIG. 1, light originating from point X₁ in the sceneis spread on an area around the theoretical image point. The lightspread around the theoretical image point of X1 is indicated by thedashed circle 34. Thus, part of the light that ideally should impinge atthe pixel 36 corresponding to the theoretical image point of X₁ actuallyimpinges at different pixels 18 of the pixel array 14, such as, forinstance the pixel 38 located at the theoretical image point of point X₂in the scene. The intensity values I₀′, I₁′, I₂′ and I₃′ (resulting fromthe charges integrated during the different integration intervals T₁,T₂, T₃ and T₄) thus differ from the ideal intensity values (not affectedby light spreading). The parameter values of A′(x), B′(x) and φ′(x) ofthe intensity waveform I′(x) reconstructed from the intensity valuesI₀′(x), I₁′(x), I₂′(x) and I₃′(x) thus differ from the ideal values ofA(x), B(x) and φ(x). This is illustrated in FIG. 2, showing theintensity waveform 32 obtained using the actually measured intensityvalues I₀′, I₁′, I₂′ and I₃′ (represented by the dashed boxes 42) andthe ideal intensity waveform (dotted curve 40) with corresponding idealintensity values represented by the dotted boxes 44. The timeindications on the time axis (horizontal axis) are expressed in units ofthe modulation period while the intensity indications on the verticalaxis are given in arbitrary units. It shall be noted that the waveforms32 and 40 not only differ in offset and amplitude but also in phase.

If the method according to the first embodiment of the invention isimplemented by 3D camera 10, the evaluation circuit 26 determines firstthe parameters A′(x) and φ′(x) for each pixel. This is illustrated asstep S10 in FIG. 3. To derive the corrected amplitudes A(x) and phasesφ(x), the evaluation circuit computes the first data array[AC′(x)]=[AC′(x₁), . . . , AC′(x_(n))] and the second data array[AS′(x)]=[AS′(x₁), . . . , AS′(x_(n))], where x₁, . . . , x_(n) standfor the pixels of the pixel array onto which part of the scene isactually imaged (step S12 in FIG. 3). It shall be noted that in thecontext of the de-convolution, the reference pixels are deemed not beingpart of the pixel array since no part of the scene is imaged on them. Itshall further be noted that the internal representation of the first andsecond data arrays in the evaluation circuit 26 can differ from therepresentation given here for illustration purposes. The values of thearray elements AC′(x) and AS′(x) may be determined as indicated before,using equations (15) and (16) or, in case of the small phaseapproximation, (17) and (18). The evaluation circuit 26 then determines(step S14 in FIG. 3) de-convoluted versions [AC(x)] and [AS(x)] of thefirst and second data arrays respectively according to

$\begin{matrix}{{{A\;{C(x)}} = {\sum\limits_{x^{\prime}}{{g^{\prime}\left( {x,x^{\prime}} \right)}{{AC}^{\prime}\left( x^{\prime} \right)}}}}{and}} & (25) \\{{{{AS}(x)} = {\sum\limits_{x^{\prime}}{{g^{\prime}\left( {x,x^{\prime}} \right)}{{AS}^{\prime}\left( x^{\prime} \right)}}}},} & (26)\end{matrix}$which corresponds to equations (19a) and (19b), taking into account thatthe integral is in this case a sum over the pixels of the pixel array.Advantageously, the de-convolution function g′ is stored in a memory ofthe evaluation circuit, e.g. in form of a matrix [[g′(x,x′)]]. Theevaluation circuit then determines (step S16 in FIG. 3), for each pixel(other than the reference pixels) the corrected phase φ(x) usingequations (21) or (22), depending on whether the small phaseapproximation is used or not. The distance values are finally computedusing the corrected phases φ(x) and the reference phase determined withthe reference pixels 19.

The method according to the first embodiment is particularly useful ifthe 3D camera gives no access to the raw data (in the above example thevalues I₀′, I₁′, I₂′ and I₃′) or if such access would be complicated.

If the method according to the second embodiment of the inventiondiscussed hereinbefore is implemented by 3D camera 10, the evaluationcircuit 26 forms data arrays [I_(k)′(x)]=[I_(k)′(x₁), . . . ,I_(k)′(x_(n))], k=0, . . . , 3. Each of these data arrays is thusassociated to a specific phase of the modulation, each array element ofthe data arrays is associated with a pixel of the pixel array and has avalue corresponding to the intensity value of the associated pixel atthe specific phase of the modulation. These data arrays are thende-convoluted, yielding de-convoluted data arrays having array elementsdefined by

$\begin{matrix}{{I_{k}(x)} = {\sum\limits_{x^{\prime}}{{g^{\prime}\left( {x,x^{\prime}} \right)}{{I_{k}}^{\prime}\left( x^{\prime} \right)}}}} & (27)\end{matrix}$which corresponds to equation (23), taking into account that theintegral is in this case a sum over the pixels of the pixel array.Alternatively, the evaluation circuit might also form data arraysobtainable from linearly combining the data arrays [I_(k)′(x)], k=0, . .. , 3 and then de-convolute these linearly combined data array, e.g. asin equation (19). In the example of FIG. 4, the arrays[I_(k)′(x)]=[I_(k)′(x₁), . . . , I_(k)′(x_(n))], k=0, . . . , 3 formedin step 20 are linearly combined to yield the data arrays[AC′(x)]=[AC′(x₁), . . . , AC′(x_(n))] and [AS′(x)]=[AS′(x₁), . . . ,AS′(x_(n))] (step 22), e.g. using the equations AS′(x)=(I₂′(x)−I₀′(x))/2and AC′(x)=(I₁′(x)−I₃′(x))/2, provided that the four phases of themodulation are spaced by 90 degrees. The evaluation circuit thendetermines (step S24 in FIG. 4) de-convoluted versions [AC(x)] and[AS(x)] of the data arrays respectively according to equation (25) and(26). The evaluation circuit then determines (step S26 in FIG. 4), foreach pixel the corrected phase φ(x) using equations (21) or (22),depending on whether the small phase approximation is used or not.

As in the previous example, the function g′ may be stored internally inthe evaluation circuit 26 e.g. in form of a matrix. The evaluationcircuit 26 than computes the corrected phases φ(x) based upon equation(21) or (22), as well as the distance values using the corrected phasesφ(x) and the reference phase determined with the reference pixels 19.

It shall be noted that those skilled in art of optical imaging systemsknow how to determine a suitable de-convolution function for a givenoptical system. Nevertheless, a specific example of a point-spreadfunction and the associated de-convolution function will now bediscussed for the purpose of illustration.

The present correction compensates the influence of the homogenous partof the stray light on the amplitude and phase measurement. Thepoint-spread function g_(h) corresponding to such homogeneous spreadingof light around the theoretical image point is given byg _(h)(x′,x)=(1−ε)δ(x′,x)+εE _(V)(x)  (28)where E_(V) denotes a function which is constant on an area V and 0elsewhere:

$\begin{matrix}{{E_{V}(x)} = {\frac{1}{\int_{V}{\mathbb{d}x}}\left\{ \begin{matrix}1 & {x \in V} \\0 & {x \notin V}\end{matrix} \right.}} & (29)\end{matrix}$

Point spread function g_(h) can be inverted. The result is

$\begin{matrix}{{{g_{h}}^{\prime}\left( {x,x^{\prime}} \right)} = {\frac{1}{1 - ɛ}\left\lbrack {{\delta\left( {x,x^{\prime}} \right)} - {ɛ\;{E_{V}\left( x^{\prime} \right)}}} \right\rbrack}} & (30)\end{matrix}$

Inserting equation (21) into (5) yields for the corrected complexamplitude the expression

$\begin{matrix}{{{\hat{A}(x)} = {\frac{1}{1 - ɛ}\left\lbrack {{\hat{A^{\prime}}(x)} - {ɛ\left\langle {\hat{A}}^{\prime} \right\rangle}} \right\rbrack}},} & (31)\end{matrix}$where <Â′> denotes the average of the complex amplitude Â′ in the areaV. This means that the correct complex phase is obtained by subtractingfrom the measured complex phase a certain portion of the average of thecomplex amplitude. The corrected phase and amplitude are then obtainedby applying equation (20) to the real and imaginary parts of Â,respectively. The de-convolution of the first and second data arrays maythus be effected through

$\begin{matrix}\begin{matrix}{{A\;{C(x)}} = {\frac{1}{1 - ɛ}\left\lbrack {{A\;{C^{\prime}(x)}} - {ɛ\left\langle {A\; C^{\prime}} \right\rangle}} \right\rbrack}} \\{{{AS}(x)} = {\frac{1}{1 - ɛ}\left\lbrack {{{AS}^{\prime}(x)} - {ɛ\left\langle {AS}^{\prime} \right\rangle}} \right\rbrack}}\end{matrix} & (32)\end{matrix}$where <AC′> and <AS′> denote the averages of AC′(x) and AS′(x),respectively, in the area V. Equation (32) expresses that thede-convolution may be effected in the case of homogeneous spreading bywithdrawing from each array element of the first data array [AC′(x)] afraction ε of an averaged value of the values of the array elements ofthe first data array and from each array element of the second dataarray [AS′(x)] the same fraction ε of an averaged value of the values ofthe array elements of the second data array.

The homogenous compensation (30) combined with the small phaseapproximation (equations (17) and (18)) yields:

$\begin{matrix}\begin{matrix}{{A(x)} = {\frac{1}{1 - ɛ}\left\lbrack {{A^{\prime}(x)} - {ɛ\left\langle A^{\prime} \right\rangle}} \right\rbrack}} \\{{{AS}(x)} = {\frac{1}{1 - ɛ}\left\lbrack {{{A^{\prime}(x)}{\varphi^{\prime}(x)}} - {ɛ\left\langle {A^{\prime}\varphi^{\prime}} \right\rangle}} \right\rbrack}} \\{{\varphi(x)} = {{{AS}(x)}/{A(x)}}}\end{matrix} & (33)\end{matrix}$

The de-convolution is easy to calculate in this manner, as it requiresonly once the computation of the mean value of the amplitudes <A′> andthe mean value of the amplitude-weighted phase (or phase-weightedamplitude) <A′φ′> in the area V. Note that the word “homogenous” refersto the stray light amplitude, not to the resulting correction termΔφ(x)=φ(x)−φ′(x). For each pixel, the correction of the phase will bestrongly depend on the phase φ′(x) and the amplitude A′(x) measured inthat pixel, and is thus not a homogeneous function. This become apparentby calculating the phase shift resulting from equation (33):

$\begin{matrix}{{{\Delta\varphi}(x)} = {\frac{{{A^{\prime}(x)}{\varphi^{\prime}(x)}} - {ɛ\left\langle {A^{\prime}\varphi^{\prime}} \right\rangle}}{{A^{\prime}(x)} - {ɛ\left\langle A^{\prime} \right\rangle}} = \frac{{{A^{\prime}\left( x^{\prime} \right)}{\varphi^{\prime}(x)}} - {ɛ\left\langle {A^{\prime}\varphi^{\prime}} \right\rangle}}{A(x)}}} & (34)\end{matrix}$

The phase shift at pixel x is thus inversely proportional to thecorrected amplitude at this pixel.

It shall be noted that the correction of homogeneous spreading canequally well be applied to the method that uses equations (23) or (19).Replacing g′(x, x′) in equation (23) by g_(h)′(x, x′) given by equation(30) yields:

$\begin{matrix}{{{{I_{k}}^{\prime}(x)} = {\frac{1}{1 - ɛ}\left\lbrack {{{I_{k}}^{\prime}(x)} - {ɛ\left\langle {I_{k}}^{\prime} \right\rangle}} \right\rbrack}},} & (35)\end{matrix}$where <I_(k)′> denotes the average of the intensity values I_(k)′(x) inthe area V. Equation (35) expresses that the data arrays associated tothe respective phases of the modulation are de-convoluted by withdrawingfrom each array element I_(k)′(x) the fraction ε of an averaged value<I′> of the values of the array elements of the data array. A similarexpression can easily be derived by replacing g′(x, x′) in equation (19)by g_(h)′(x, x′).

The parameter ε quantifies the amount of light that is homogenouslystrayed by the optics of the system. Usually this parameter can bedetermined by an optical measurement and takes a fixed value for a givenoptics. However, in case of contamination of the optical system, theamount of scattered light can increase. If the level of contamination isdynamically determined and the parameter ε adjusted in accordance, thepresent method also allows correcting the loss of phase and amplitudecontrast on account of the level of contamination. A method fordetermining the level of contamination is disclosed, for instance, inEuropean patent application 07 110 379.0, which is herewith incorporatedherein by reference in its entirety.

The invention claimed is:
 1. Method of recording 3D images of a scenebased on a time-of-flight principle, comprising illuminating a scene byemitting light carrying an intensity modulation; imaging the scene ontoa pixel array using an optical system; detecting, in each pixel,intensity-modulated light reflected from the scene onto said pixel, saidintensity-modulated light detected in the pixel having an amplitude anda phase; determining, for each pixel, an amplitude value and a phasevalue of the amplitude and the phase, respectively , of saidintensity-modulated light detected in the pixel determining, for eachpixel, a distance value based on the phase value of saidintensity-modulated light detected in the pixel; wherein determiningsaid distance values comprises a phase-sensitive de-convolution of saidscene imaged onto said pixel array such as to compensate for phaseerrors induced by light spreading in said optical system, wherein saidphase-sensitive de-convolution of said scene imaged onto said pixelarray comprises forming a first data array, each array element of saidfirst data array being associated with a pixel of said pixel array andhaving a value corresponding to the amplitude value determined for theassociated pixel weighted with a first phase factor; forming a seconddata array, each array element of said second data array beingassociated with a pixel of said pixel array and having a valuecorresponding to the amplitude value determined for the associated pixelweighted with a second phase factor, said second phase factor dependingon the phase value determined for the associated pixel; de-convolutingsaid first and second arrays based upon a de-convolution function ofsaid optical system, wherein, for each pixel, said distance value iscalculated based upon the values of the array elements of saidde-convoluted first and second arrays associated to the pixel, andwherein the de-convolution of said first and second arrays is effectedby withdrawing from each array element of said first array a certainfraction of an averaged value of the values of the array elements ofsaid first array and from each array element of said second array acorresponding fraction of an averaged value of the values of the arrayelements of said second array.
 2. The method according to claim 1,wherein said first phase factor is the cosine of the phase valuedetermined for the associated pixel and wherein said second phase factoris the sine of the phase value determined for the associated pixel. 3.The method according to claim 2, wherein, for each pixel, calculatingthe distance value comprises determining a corrected phase asφ(x)=arctan(AS(x)/AC(x)), where x identifies the pixel, φ(x) denotes thecorrected phase, AC(x) denotes the array element of the de-convolutedfirst array associated to the pixel and AS(x) denotes the array elementof the de-convoluted second array associated to the pixel.
 4. The methodaccording to claim 1, wherein said first phase factor is 1 and whereinsaid second phase factor is the phase value itself.
 5. The methodaccording to claim 4, wherein for each pixel, calculating the distancevalue comprises determining a corrected phase asφ(x)=AS(x)/AC(x), where x identifies the pixel, φ(x) denotes thecorrected phase, AC(x) denotes the array element of the de-convolutedfirst array associated to the pixel and AS(x) denotes the array elementof the de-convoluted second array associated to the pixel.
 6. The methodaccording to claim 1, wherein a level of contamination of said opticalsystem is evaluated and wherein said phase-sensitive de-convolution isadjusted to said level of contamination.
 7. Method of recording 3Dimages of a scene based on a time-of-flight principle, comprisingilluminating a scene by emitting light carrying an intensity modulation;imaging the scene onto a pixel array using an optical system; detecting,in each pixel, intensity-modulated light reflected from the scene ontosaid pixel, said intensity-modulated light detected in the pixel havingan amplitude and a phase, said detecting of intensity-modulated lightreflected from said scene comprising, for each pixel, determiningintensity values of the intensity-modulated light impinging on the pixelat different modulation phases, said different modulation phases beingchosen such that amplitude and phase of the intensity-modulated lightimpinging on said pixel are derivable from said set of intensity valuesusing a known relationship; determining, for each pixel, a distancevalue based on a phase-sensitive de-convolution of said scene imagedonto said pixel array such as to compensate for phase errors induced bylight spreading in said optical system, wherein said phase-sensitivede-convolution of said scene comprises forming data arrays, each arrayelement of said data arrays being associated with a pixel of said pixelarray and having a value corresponding either to the intensity value ofthe associated pixel determined at one of said modulation phases or to alinear combination of at least two intensity values of the associatedpixel determined at different modulation phases; de-convoluting saiddata arrays using a de-convolution function of said optical system,wherein, for each pixel, the distance value is calculated based upon thevalues of the array elements of said de-convoluted data arraysassociated to the pixel, and wherein the de-convolution of each of saiddata arrays is effected by withdrawing from each array element of thedata array a certain fraction of an averaged value of the values of thearray elements of the data array.
 8. The method according to claim 7,wherein said intensity values are determined at least three differentmodulation phases.
 9. The method according to claim 8, wherein said atleast three modulation phases are regularly spaced.
 10. The methodaccording to claim 9, wherein said intensity values are determined atfour modulation phases, said four modulation phases being spaced by 90degrees.
 11. The method according to claim 7, wherein, for each pixel,calculating said distance value comprises determining a corrected phasefrom the values of the array elements of said de-convoluted data arraysassociated to the pixel.
 12. The method according to claim 7, wherein alevel of contamination of said optical system is evaluated and whereinsaid phase-sensitive de-convolution is adjusted to said level ofcontamination.
 13. A 3D time-of-flight imager, comprising a light sourceconfigured to illuminate a scene by emitting light carrying an intensitymodulation; a pixel array; an optical system configured to image thescene onto said pixel array, each pixel of said pixel array beingconfigured to detect intensity-modulated light reflected from the sceneonto said pixel, said intensity-modulated light detected in the pixelhaving an amplitude and a phase; and a control and evaluation circuitconfigured to determine, for each pixel, a distance value based on thephase of said intensity-modulated light detected in the pixel; whereinsaid control and evaluation circuit is configured to carry out aphase-sensitive de-convolution of said scene imaged onto said pixelarray in such a way as to compensate for phase errors induced by lightspreading in said optical system, wherein said phase-sensitivede-convolution of said scene imaged onto said pixel array comprisesforming a first data array, each array element of said first data arraybeing associated with a pixel of said pixel array and having a valuecorresponding to the amplitude value determined for the associated pixelweighted with a first phase factor; forming a second data array, eacharray element of said second data array being associated with a pixel ofsaid pixel array and having a value corresponding to the amplitude valuedetermined for the associated pixel weighted with a second phase factor,said second phase factor depending on the phase value determined for theassociated pixel; de-convoluting said first and second arrays based upona de-convolution function of said optical system, wherein, for eachpixel, said distance value is calculated based upon the values of thearray elements of said de-convoluted first and second arrays associatedto the pixel, and wherein the de-convolution of said first and secondarrays is effected by withdrawing from each array element of said firstarray a certain fraction of an averaged value of the values of the arrayelements of said first array and from each array element of said secondarray a corresponding fraction of an averaged value of the values of thearray elements of said second array.
 14. The 3D time-of-flight imageraccording to claim 13, wherein said control and evaluation circuitcomprises at least one of an application-specific integrated circuit, afield-programmable gate array and a microprocessor to carry out saidphase-sensitive de-convolution of said scene imaged onto said pixelarray.